Abstract Title Page 


Title: 

Replicating the Relationship Between Teachers’ Data Use and Student Achievement: The Urban 
Data Study and the Data Dashboard Usage Study 

Authors and Affiliations: 

Ann-Marie Faria 
Ariela Greenberg 
John Meakin 
Krystal Bichay 
Jessica Heppen 

American Institutes for Research 


SREE Spring 2013 Conference Abstract 



Abstract Body 


Background/ Context : 

Educators have long used test scores to make educational decisions, but only within the last 
decade has the availability of data been systematic (Abelman, Elmore, Even, Kenyon, & 
Marshall, 1999). In recent years, interest has spiked in data-driven decision making in education 
(Marsh, Pane, & Hamilton, 2006). With technological advances and growing pressure for “data- 
driven” instruction under accountability reforms, many schools, districts, and states have 
invested significant resources in tools designed to provide educators with regular access to data 
(U.S. Department of Education, 2009a; 2009b; 2009c). The increased emphasis on data use is 
based on the belief that student data can be an important lever for improved teaching and 
learning. Although the relationship between data use and student achievement has face validity, 
little empirical evidence links educators’ use of data with improved student achievement. 

One recent study examined the relationship between key dimensions of interim assessment data 
use and student achievement. The study, known as the “Urban Data Study (UDS),” was funded 
by the Bill & Melinda Gates Foundation and conducted by American Institutes for Research and 
the Council for the Great City Schools. The study examined links between educators’ data use 
(measured by teacher self-reported surveys) and student achievement on end-of-year state 
assessments. This study contributed to the field by defining a conceptual framework for the 
effective use of data (see Appendix B, Figure 1, for the theory of action that links data use with 
student achievement), creating new measures of data use practices and providing some early 
evidence of a relationship between data use and achievement by empirically connecting data use 
practices to changes in teaching and actual improvements in student outcomes. 

With initial findings from the UDS suggesting an empirical link between data use and student 
achievement, the next step was to refine the measurement strategy and expand the type of data 
examined during educators’ data use. The next study in this line of research is the Student 
Dashboard Usage Study, funded by a different foundation. The study examined educators’ use of 
Web-based data dashboard system and employed some of the same methods as the UDS as well 
as some innovative measurement techniques. Although we again collected self-reported survey 
data from teachers about their data use, we also capitalized on the increasing availability of Web- 
log system usage data. Such usage data can provide an objective way to gauge the frequency, 
duration, and type of data review activities in which educators engage when using an online data 
system. 

In line with the theme for the spring 2013 Society for Research on Educational Effectiveness 
(SREE) conference, the current study attempts to replicate the UDS findings using the new data 
from the Data Dashboard Usage Study. The current study will compare the initial findings from 
the UDS with new findings from the Data Dashboard Usage Study regarding potential links 
between educators’ data use and their students’ achievement. 

Purpose/Objective/Research Question/Focus of Study: 

The overarching goal of the current study is to compare and contrast findings between the UDS 
and the Student Dashboard Usage Study. In this project, we attempt to replicate findings from 
the UDS (Faria et al., 2012) to further understand the links between teacher-level data-use 
practices and student achievement. Both the UDS and the Student Dashboard Usage Study 
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answer the broad research question: What are the relationships between teachers ’ data-use 
practices and their students ’ achievement? By examining the findings from both studies 
together, we will better understand the relationship between data use and student achievement. 

Setting: 

Although both studies were conducted in multiple districts and statically modeled the 
relationship between teacher data use and student achievement on state tests, some key 
differences between the two studies should be noted. Table 1 in Appendix B provides a 
breakdown of district characteristics for each study. We conducted the UDS in four 
geographically diverse urban districts across the United States. Inclusion in the study required 
that districts had administered interim assessments at least three times per school year continuously 
for the past three years and planned to continue administering them for at least the next several 
years and had a data system with the capacity to meet the requirements of a quantitative study 
that would link school- and classroom-level data use practices with student achievement. District 
size ranged from one of the largest urban districts in the nation to a mid-sized district. The 
Student Dashboard Usage Study included five diverse districts in one southwestern state, 
strategically chosen by the foundation because of their demographic characteristics, location, and 
willingness to participate as pilot districts for the state’s new data dashboard initiative. 

Population/Participants/Subjects: 

The UDS included teachers of Grades 4, 5, 7, or 8 (n = 1,581) in a sample of 193 randomly 
selected elementary and middle schools in the four participating districts. The analyses linked 
teacher and principal survey data about their own data use with a total of 61,798 students in those 
grade levels. The Student Dashboard Usage Study will include two samples. The first includes 
1,361 teachers who used the online data dashboard and taught Grades 4-12 in reading or 
mathematics in the 2012-2013 academic year. These teachers were linked with more than 50,000 
students. The second sample includes 561 randomly selected teachers who completed the self- 
reported data use survey and teach reading and mathematics. These 561 teachers were linked 
with just over 25,000 students. 

Intervention/Program/Practice : 

Neither study implemented an intervention, but each studied perceptions and practices related to 
data-use initiatives being implemented by each district. UDS focused specifically on educators’ 
use of district-specific interim assessments and examined the link between educators’ use of 
interim assessments and student achievement on end-of-year state tests in mathematics and 
reading. The breadth of the data use in the Student Dashboard Usage Study was more 
comprehensive than in the UDS; that is, the dashboards encompassed different types of data, 
including interim assessment data (i.e., the focus of the UDS) but also included attendance, 
behavior, grades, credits, prior achievement, student contact information, and background 
characteristics. Also, the Student Dashboard Usage Study will use both educators’ self-reported 
data use by way of teacher surveys and Web-log data that captures real-time review of the data 
dashboards. This study empirically tests the link between teachers’ data use of the more 
comprehensive data dashboards with student achievement on end-of-year state tests in 
mathematics and reading. 
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Research Design: 

Both the UDS and the Student Dashboard Usage Study used a multilevel correlational design to 
empirically test links between teacher-level data use and student achievement. Neither study 
attempted to model a causal relationship between data use and student achievement. The 
proposed paper will examine the extent to which the findings from the UDS can be replicated 
with a new study that used some common measures of data use practices and perceptions 
(teacher surveys) and some new measures based on Web-log data of actual data use. 

Data Collection and Analysis: 

The study team had access to three types of data for these projects that are relevant to the 
proposed paper. In both studies, we administered multiple waves of teacher surveys about data 
use and collected district records, including student-level achievement scores and demographic 
information as well as class rosters that enabled us to link students to their teachers. In the 
Student Dashboard Usage Study, Web-log usage data were also available. 

Teacher and principal surveys. In both studies, we developed surveys to measure key concepts in 
the theory of action (see Figure 1), including the frequency with which data are used and the 
manner and extent to which data inform instruction and decision making. 1 For subscales of data 
use, internal consistency was adequate with Cronbach alphas ranging from 0.70 to 0.93 for 
teachers and 0.76 to 0.97 for principals. The survey also included measures of teacher 
background characteristics, used as control variables in the analyses. 

District records data. In both studies, we collected individual student-level demographic and 
achievement data from district databases. Our primary outcome measures are state assessment 
scores in reading or English and mathematics. In the UDS, these were scores from spring 2010, 
and in the Student Dashboard Usage Study, they were scores from spring 2013. In both studies, 
student characteristics and prior achievement were used as control variables in the analyses. In 
both studies, we used class rosters for reading or English and mathematics teachers to link 
teachers in the survey sample to the individual records for the students in their classrooms. 

Web-log data. In the Student Dashboard Usage Study, we also received data culled directly from 
the data dashboard Web-logs that tracked each user’s weekly activity. The data allowed us to 
create measures reflecting four main aspects of usage: log-ins, duration associated with use, page 
views, and actions. “Actions” refer to interactions with pages that did not direct users to another 
page. These four main aspects of usage were also used as predictors of student achievement in 
the Student Dashboard Usage Study. 

For both studies, we used two-level hierarchical linear models that adjust for the nested nature of 
the data, with students at Level 1 and teachers at Level 2. All models control for key student 
covariates, such as prior achievement, gender, free or reduced-price lunch, and minority status. 
Models also include classroom-level aggregates of prior achievement as a control variable to 
increase the precision of the estimates of the relationship between data use and student 
achievement. 


1 The paper will provide additional information about instrument development. 
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Key differences between the models are that the UDS focused only on Grades 4, 5, 7, and 8, 
whereas the Student Dashboard Usage study included teachers of Grades 4 through 12. Although 
the key predictors in the UDS are related to teachers’ self-reported data use practices, data from 
the Student Dashboard Usage Study allow us to conduct two sets of analyses. The first set of 
analyses will examine possible links between teachers’ self-reported data use and student 
achievement (the closest replication of the UDS findings). The second set of analyses examines 
possible links between teachers’ actual data use based on the Web-log files and student 
achievement. To date, all analyses are complete for the UDS. Analyses are currently under way 
for the Student Dashboard Usage Study and will be completed well in advance of the spring 
2014 SREE conference. 

Findings/Results : 

In the UDS, previously presented at SREE in 2013, hierarchical linear modeling results 
suggested significant links between some key dimensions of teacher data use and student 
achievement (see Table 2 in Appendix B). Specifically, in middle grades mathematics and 
elementary grades reading, Attention to Data in the Classroom was significantly and positively 
related to student achievement /? = 0.09, p < 0.01 and /? = 0.06, p < 0.05, respectively. This key 
dimension was a compound construct that included the measures of Working with Data and 
Instructional Responses. Though not consistent across grades and subjects, Attention to Data in 
the Classroom was the strongest predictor of student achievement. Teacher perceptions of 
Barriers to using data also predicted lower student achievement in elementary mathematics, fi = 
-0.08, p < 0.05. In combination, these findings partially supported the theory of action. For the 
proposed paper, we will replicate the models described previously using the more up-to-date and 
comprehensive data from the Student Dashboard Usage Study to determine whether the relationship 
between educators’ use of data and student achievement are robust across the two studies. 

Conclusions: 

In the UDS, we hypothesized that specific data-use practices and perceptions would be positively 
related to student achievement. The findings partially supported this hypothesis. Teachers’ 
Attention to Data in the Classroom were related to higher student achievement, whereas 
teachers’ perceived data use Barriers were related to lower student achievement. In other words, 
the more that teachers and principals reported reviewing and analyzing student data and using 
this information to make instructional decisions, the higher their students’ achievement, at least 
in some grades and subjects. 

However, the findings varied by grade and content area, with significant links observed in both 
elementary grades and middle grades, as well as in mathematics and reading. To better 
understand the nature of the relationships between data use and student achievement, we will 
attempt to replicate these findings with the new data from the Student Dashboard Usage Study. 
Observations of similar patterns with data from the Student Dashboard Usage Study would lend 
confidence in the specific relationships found in the UDS. The paper and the presentation will 
discuss the possible reasons for consistency, or lack thereof, and will contribute to the audience 
and the field a set of next questions about this highly relevant policy issue concerning data- 
driven decision making in school settings. 
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Appendix B. Tables and Figures 


Table 1: District Characteristics by Study 



Total Number of 
Schools 

Total Number of 
Teachers 

Total Number of 
Students 

Urban Data Study 

District 1 

126 

6,500 

90,000 

District 2 

324 

14,800 

311,000 

District 3 

155 

6,000 

98,000 

District 4 

53 

1,900 

24,000 

Student Dashboard Usage Study 

District A 

22 

1,090 

16,500 

District B 

7 

220 

3,000 

District C 

54 

1,900 

29,000 

District D 

44 

1,840 

32,000 

District E 

71 

3,800 

51,000 


Table 2: Key Findings from the Urban Data Study. 



Mathematics 


Elementary Grades 

Middle Grades 

Teacher Data Use Scales 

(P) 

(P) 

Context 

0.03 

0.04 

Supports for Data Use 

0.01 

-0.01 

Attention to Data in the Classroom 

0.04 

0.09** 

Barriers to Data Use 

-0.08 * 

-0.04 


Reading 


Elementary Grades 

Middle Grades 


(P) 

(P) 

Context 

0.03 

0.06 

Supports for Data Use 

-0.01 

0.01 

Attention to Data in the Classroom 

0.06* 

0.02 

Barriers to Data Use 

-0.02 

0.00 


Note. *p < 0.05; ** p < 0.01 (two-tailed) 
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Figure 1. Theory of Action, Revised Based on Findings From the UDS and Used to Inform Data Collection in the Student 

Dashboard Usage Study 
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